基于海明距离的加权平均值人职匹配模型(Sqlserver2014/16内存表实现)
最近给某大学网站制作一个功能,需要给全校所有的学生提供就业单位发布职位的自动匹配,学生登陆就业网,就可以查看适合自己的职位,进而可以在线投递。
全校有几万名学生,注册企业发布的职位也有上万,如何在很短的时间内(不影响学生访问网站),通过建立好的匹配模型迅速的对学生——职位进行匹配?
- 建模篇
我以前给银行开发过房地产自动估价软件,按照标准做法是用欧几里得贴近算法或者海明贴近度,但是那种算法太复杂,属于应用数学的范畴,需要依靠精准的建模。我们就业办的老师是在实战上打拼的,没有高深的理论基础,所以建模必须简单。根据调研,发现使用海明距离的结合加权的算法比较简单,也容易转化成匹配度的百分比。具体算法出自2010年3月《计算机工程》中的《模糊匹配中的匹配度计算方法》:
这种算法的基本思路是每个项目都有一定的权重分值,然后按照学生和职位每个项目的匹配度,乘以权重分值再除以总的权重,就是学生对于每个职位的匹配情况。
学生在就业网上设置过职位搜索器的,按照职位搜索器中设置的项目进行匹配,有这么些项目:
比如“薪资”一项的匹配方法如下:
如果学生没有设置过搜索器的,则使用另一套维度,其中使用了就业数据的大数据分析方法,介绍略。
- 实现篇
网站基于.net C# + Sqlserver,每次要对所有的有效职位进行匹配,如此大的计算量使用传统方法肯定会慢,甚至造成性能瓶颈。因此思考采用两种方式提高计算效率:
- 使用MongoDB。
- 使用Sqlserver2014或者SqlServer2016的内存优化表。
MongoDB是典型的NoSQL数据库,交换数据是json格式,这种数据库存取的速度非常快,没有Sqlserver那些复杂的权限、并发、锁、存储引擎,因此很适合作为高吞吐量的数据存储方式;
微软在Sqlserver2014和Sqlserver2016中开发了内存优化表和本地编译存储过程,两者也有很好的性能表现
(顺便吐槽一下网上有人说Oracle一句命令就可以把表升级为内存表,一句命令就可以把存储过程升级成本地编译存储过程,而Sqlserver这方面限制太多,内存表不能建索引、不能建Check....(2016版可以),而本地编译存储过程的限制更是多得多,不能用function,不能用游标,不能用链接数据库........。我想这是两种数据库不同的实现机制形成的,在《SQL编程风格》151页中描述:“T-SQL是一个简单的一遍扫描的编译器,以C和Algol语言模型创建......Oracle中的PL/SQL是以ADA和SQL/PSM为模型创建的,它是一种复杂语言,可以用来开发应用程序。”所以Oralce的存储过程要升级简直易如反掌。)
我们还是先选择SqlServer的内存表作为数据缓冲池,本来想使用本地编译存储过程实现模型匹配算法,但是限制实在太多,所以只好使用普通的存储过程。每天有批处理把职位数据同步到内存表里,然后学生登录后进行计算,每周还进行职位的推送。
应大家要求公布算法代码:(因为和学校签订过保密协议,所以删除部分行,请谅解)
基本的建表、初始化数据脚本:
--建表
CREATE DATABASE [DataAnalysis]
CONTAINMENT = NONE
ON PRIMARY
( NAME = N'DataAnalysis', FILENAME = N'd:\DATA\DataAnalysis.mdf' , SIZE = 5120KB , MAXSIZE = UNLIMITED, FILEGROWTH = 1024KB ),
FILEGROUP [DataAnalysisFileGroup] CONTAINS MEMORY_OPTIMIZED_DATA DEFAULT
( NAME = N'DataAnalysisContainer', FILENAME = N'd:\DATA\HashCollisionsContainer' , MAXSIZE = UNLIMITED)
LOG ON
( NAME = N'DataAnalysis_log', FILENAME = N'd:\DATA\DataAnalysis_log.ldf' , SIZE = 2304KB , MAXSIZE = 2048GB , FILEGROWTH = 10%)
GO ALTER DATABASE [DataAnalysis] SET COMPATIBILITY_LEVEL = 120
GO Use [DataAnalysis]
GO if exists(select * from sysobjects where id=object_id('EnterPrisePositions'))
DROP TABLE EnterPrisePositions
GO CREATE TABLE EnterPrisePositions
(
[ID] [int] IDENTITY(1,1) NOT NULL Primary Key NONCLUSTERED HASH WITH (BUCKET_COUNT = 4096),
[EntID] uniqueidentifier NOT NULL,
[EntUserID] uniqueidentifier NOT NULL,
[EntName] [nvarchar](80) NOT NULL,
[PosiID] [uniqueidentifier] NOT NULL,
[PosiName] [nvarchar](40) NULL,
[JobTypeID] [uniqueidentifier] NULL,
--.......
) WITH (MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_ONLY)
GO INSERT INTO [EnterPrisePositions] ([EntID], [EntUserID], [EntName], [OrgCode]..........)
SELECT e.[EntID], p.[EntUserID], e.[EntName], e.[OrgCode], e.[HyID], e.[SubHyID], e.[ThirdHyID],..........
FROM .....[dbo].[Position] p INNER JOIN ......[dbo].[Enterprise] e ON p.[EntUserID] = e.[EntUserID]
WHERE p.DelFlag <> 1 AND p.EffectiveDate <= GetDate() AND GetDate() <= p.ExpiryDate AND e.[CheckFlag] = 1 AND e.[DelFlag] = 0 AND e.[IsBlack] = 0
GO if exists(select * from sysobjects where id=object_id('UserPositionResult'))
DROP TABLE UserPositionResult
GO CREATE TABLE UserPositionResult
(
[ID] [int] IDENTITY(1,1) NOT NULL Primary Key NONCLUSTERED HASH WITH (BUCKET_COUNT = 4096),
[Score] FLOAT,
[PosiID] [uniqueidentifier] NOT NULL,
[PosiName] [nvarchar](40) NULL,
--............
) WITH (MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_ONLY)
GO --辅助函数
-- =============================================
-- Author: Ben
-- Create date: 2016-08-08
-- Description: 处理专业ID用
-- =============================================
CREATE FUNCTION [GetPositionSpecialty] (
@String NVARCHAR(150)
) RETURNS @temptable TABLE (ID INT IDENTITY(1,1), Specialty NVARCHAR(8)) AS
BEGIN DECLARE @idx INT=1
DECLARE @slice NVARCHAR(150)
DECLARE @Delimiter NVARCHAR(1) = ','
IF LEN(@String) < 1 OR LEN(ISNULL(@String,'')) = 0
BEGIN
INSERT INTO @temptable(Specialty) VALUES('')
RETURN
END
WHILE @idx != 0
BEGIN
SET @idx = CHARINDEX(@Delimiter,@String)
IF @idx != 0
SET @slice = LEFT(@String,@idx - 1)
ELSE
SET @slice = @String
IF LEN(@slice) > 0
INSERT INTO @temptable(Specialty) VALUES(@slice)
SET @String = RIGHT (@String, LEN(@String) - @idx)
IF LEN(@String) = 0
BREAK
END
RETURN
END GO -- =============================================
-- Author: Ben
-- Create date: 2016-08-08
-- Description: 处理职位搜索器用
-- =============================================
CREATE PROCEDURE [GetStudentSearch]
-- Add the parameters for the stored procedure here
@Ssqtj NVARCHAR(1000), @ProvinceId NCHAR(6) OUTPUT, @Zydm NVARCHAR(10) OUTPUT, @JobNature NVARCHAR(8) OUTPUT, @JobTypeID uniqueidentifier OUTPUT, @SubJobTypeID uniqueidentifier OUTPUT,
@Salary NVARCHAR(30) OUTPUT, @Computer NVARCHAR(50) OUTPUT, @Language NVARCHAR(50) OUTPUT, @Education NVARCHAR(4) OUTPUT, @HyID uniqueidentifier OUTPUT, @SubHyID uniqueidentifier OUTPUT,
@ThirdHyID uniqueidentifier OUTPUT
AS
BEGIN
DECLARE @idx INT=1, @StartPos INT
DECLARE @Delimiter NVARCHAR(1) = '|' IF LEN(@Ssqtj) < 1 OR LEN(ISNULL(@Ssqtj, '')) = 0
RETURN SET @idx = CHARINDEX(@Delimiter, @Ssqtj, @idx)
IF @idx != 0
SET @ProvinceId = LEFT(@Ssqtj, @idx - 1) SET @StartPos = @idx + 1
SET @idx = CHARINDEX(@Delimiter, @Ssqtj, @StartPos)
IF @idx != 0
SET @Zydm = SUBSTRING(@Ssqtj, @StartPos, @idx - @StartPos) SET @StartPos = @idx + 1
SET @ThirdHyID = SUBSTRING(@Ssqtj, @StartPos, 36)
END TRY
BEGIN CATCH
END CATCH
END
GO --重要,参数表
if exists(select * from sysobjects where id=object_id('Parameters'))
DROP TABLE Parameters
GO
CREATE TABLE [Parameters]
(
[ID] [int] IDENTITY(1,1) NOT NULL Primary Key NONCLUSTERED HASH WITH (BUCKET_COUNT = 1024),
[Type] NVARCHAR(50) NOT NULL,
[Set1] NVARCHAR(50) NULL,
[Set2] NVARCHAR(50) NULL,
[Value] FLOAT NOT NULL
) WITH (MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_AND_DATA)
GO INSERT INTO [Parameters] ([Type], [Set1], [Set2], [Value])
SELECT 'Salary', '面议', '面议', 10
UNION ALL
SELECT 'Salary', '面议', '1500~2000', 10
UNION ALL
SELECT 'Salary', '面议', '2000~3000', 10
UNION ALL
SELECT 'Salary', '面议', '3000~4000', 10
UNION ALL
SELECT 'Salary', '面议', '4000~5000', 10
UNION ALL
SELECT 'Salary', '面议', '5000~6000', 10
UNION ALL
SELECT 'Salary', '面议', '6000~7000', 10
UNION ALL
SELECT 'Salary', '面议', '7000~8000', 10
UNION ALL
SELECT 'Salary', '面议', '8000以上', 10 UNION ALL
SELECT 'Salary', '1500~2000', '面议', 10
UNION ALL
SELECT 'Salary', '1500~2000', '1500~2000', 10
UNION ALL
SELECT 'Salary', '1500~2000', '2000~3000', 5
UNION ALL
SELECT 'Salary', '1500~2000', '3000~4000', 2
UNION ALL
SELECT 'Salary', '1500~2000', '4000~5000', 2
UNION ALL
SELECT 'Salary', '1500~2000', '5000~6000', 2
UNION ALL
SELECT 'Salary', '1500~2000', '6000~7000', 4
UNION ALL
SELECT 'Salary', '1500~2000', '7000~8000', 3
UNION ALL
SELECT 'Salary', '1500~2000', '8000以上', 2 UNION ALL
SELECT 'Salary', '2000~3000', '面议', 10
UNION ALL
SELECT 'Salary', '2000~3000', '1500~2000', 10
UNION ALL
SELECT 'Salary', '2000~3000', '2000~3000', 10
UNION ALL
SELECT 'Salary', '2000~3000', '3000~4000', 6
UNION ALL
SELECT 'Salary', '2000~3000', '4000~5000', 5
UNION ALL
SELECT 'Salary', '2000~3000', '5000~6000', 4
UNION ALL
SELECT 'Salary', '2000~3000', '6000~7000', 5
UNION ALL
SELECT 'Salary', '2000~3000', '7000~8000', 4
UNION ALL
SELECT 'Salary', '2000~3000', '8000以上', 3 UNION ALL
SELECT 'Salary', '3000~4000', '面议', 10
UNION ALL
SELECT 'Salary', '3000~4000', '1500~2000', 10
UNION ALL
SELECT 'Salary', '3000~4000', '2000~3000', 10
UNION ALL
SELECT 'Salary', '3000~4000', '3000~4000', 10
UNION ALL
SELECT 'Salary', '3000~4000', '4000~5000', 8
UNION ALL
SELECT 'Salary', '3000~4000', '5000~6000', 6
UNION ALL
SELECT 'Salary', '3000~4000', '6000~7000', 6
UNION ALL
SELECT 'Salary', '3000~4000', '7000~8000', 5
UNION ALL
SELECT 'Salary', '3000~4000', '8000以上', 4 --........ UNION ALL
SELECT 'Weight', 'HasNoSearch', 'Education', 20
UNION ALL
SELECT 'Weight', 'HasNoSearch', 'Profession', 20
UNION ALL
SELECT 'Weight', 'HasNoSearch', 'Industry', 8
UNION ALL
SELECT 'Weight', 'HasNoSearch', 'Enterprise', 12
GO --其他临时表,大数据分析
CREATE TABLE IndutryRanking
(
Gzydm NVARCHAR(10) COLLATE Chinese_PRC_Stroke_90_BIN2 NOT NULL, SubIndustry uniqueidentifier NOT NULL, Ranking TINYINT NOT NULL,
CONSTRAINT [PK_IndutryRanking] PRIMARY KEY NONCLUSTERED HASH
(
Gzydm ,
SubIndustry
)WITH ( BUCKET_COUNT = 2048) CREATE TABLE EnterpriseRanking
(
Gzydm NVARCHAR(10) COLLATE Chinese_PRC_Stroke_90_BIN2 NOT NULL, Zzjgdm NVARCHAR(10) COLLATE Chinese_PRC_Stroke_90_BIN2 NOT NULL, Ranking FLOAT NOT NULL,
CONSTRAINT [PK_EnterpriseRanking] PRIMARY KEY NONCLUSTERED HASH
(
Gzydm ,
Zzjgdm
)WITH ( BUCKET_COUNT = 131072)
) WITH (MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_ONLY)
GO INSERT INTO EnterpriseRanking (Gzydm, Zzjgdm, Ranking)
SELECT Gzydm, Zzjgdm, LOG(Count(*)+1, 1.8)
FROM ..........[dbo].[vm_AllSyEmployment]
WHERE Gzydm IS NOT NULL AND ZZJGDM IS NOT NULL
GROUP BY Gzydm,Zzjgdm
ORDER BY Count(*) DESC GO CREATE FUNCTION [dbo].[GetCurrentBynf]()
RETURNS char(4)
AS
BEGIN
-- Declare the return variable here
DECLARE @Bynf int -- Add the T-SQL statements to compute the return value here
SELECT @Bynf = YEAR(GETDATE()) IF MONTH(GETDATE()) >= 9 OR ( MONTH(GETDATE()) = 8 AND DAY(GETDATE()) >25 )
BEGIN
SET @Bynf = @Bynf + 1
END RETURN CAST(@Bynf AS char(4))
END GO
做个批处理,每次重启数据库的时候把数据加入,每天定时更新相关数据表
职位匹配脚本:
CREATE PROCEDURE RetirePositionsByXsxh
--参数
-- Add the parameters for the stored procedure here
@Xsxh NVARCHAR(20), @ResultType INT = 0, @PageSize INT = 99999, @StartPage INT = 0, @ReleaseDateRange SMALLINT = 9999
WITH ENCRYPTION
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON; --声明变量
-- Insert statements for procedure here
DECLARE @Ssqtj NVARCHAR(1000), @TableRows int, @PositionRows int,
@ProvinceId NCHAR(6), @Zydm NVARCHAR(10), @JobNature NVARCHAR(8), @JobTypeID uniqueidentifier, @SubJobTypeID uniqueidentifier,@Salary NVARCHAR(30),
@Computer NVARCHAR(50), @Language NVARCHAR(50), @Education NVARCHAR(4), @HyID uniqueidentifier, @SubHyID uniqueidentifier, @ThirdHyID uniqueidentifier,
--........ DECLARE @StudentSsqtj TABLE (SsqtjID INT IDENTITY(1,1) NOT NULL Primary Key, Ssqtj NVARCHAR(1000))
DECLARE @PositionSpecialty TABLE (ID INT NOT NULL Primary Key, Specialty NVARCHAR(1000))
--........... INSERT INTO @StudentSsqtj (Ssqtj)
SELECT Ssqtj
FROM .......[dbo].[PosiSearch] WITH (SNAPSHOT)
WHERE Xsxh = @Xsxh SELECT @TableRows = Count(*) FROM @StudentSsqtj
SELECT @PositionRows = Count(*) FROM [EnterPrisePositions] IF @TableRows > 0
BEGIN
SELECT @Salary_Weight = [Value] FROM [Parameters] WITH (SNAPSHOT) WHERE [Type] = 'Weight' AND [Set1] = 'HasSearch' AND [Set2] = 'Salary'
SELECT @JobLocation_Weight = [Value] FROM [Parameters] WITH (SNAPSHOT) WHERE [Type] = 'Weight' AND [Set1] = 'HasSearch' AND [Set2] = 'JobLocation'
SELECT @Education_Weight = [Value] FROM [Parameters] WITH (SNAPSHOT) WHERE [Type] = 'Weight' AND [Set1] = 'HasSearch' AND [Set2] = 'Education'
--........ SELECT @JobLocation_Same_Province = [Value] FROM [Parameters] WITH (SNAPSHOT) WHERE [Type] = 'Province' AND [Set1] = 'SameProvince'
SELECT @JobLocation_Same_City = [Value] FROM [Parameters] WITH (SNAPSHOT) WHERE [Type] = 'Province' AND [Set1] = 'SameCity'
SELECT @JobLocation_Not_Same = [Value] FROM [Parameters] WITH (SNAPSHOT) WHERE [Type] = 'Province' AND [Set1] = 'NotSame'
SELECT @Education_Same = [Value] FROM [Parameters] WITH (SNAPSHOT) WHERE [Type] = 'Education' AND [Set1] = 'Same'
SELECT @Education_Not_Same = [Value] FROM [Parameters] WITH (SNAPSHOT) WHERE [Type] = 'Education' AND [Set1] = 'NotSame'
--.......... WHILE @TableRows != 0
BEGIN
SELECT @Ssqtj = Ssqtj
FROM @StudentSsqtj
WHERE SsqtjID = @TableRows --取职位搜索器
EXEC [GetStudentSearch] @Ssqtj, @ProvinceId OUTPUT, @Zydm OUTPUT, @JobNature OUTPUT, @JobTypeID OUTPUT , @SubJobTypeID OUTPUT, @Salary OUTPUT,
@Computer OUTPUT, @Language OUTPUT, @Education OUTPUT, @HyID OUTPUT, @SubHyID OUTPUT,@ThirdHyID OUTPUT WHILE @PositionRows != 0
BEGIN --取每个职位
SELECT @Pos_HyID = [HyID], @Pos_SubHyID = [SubHyID], @Pos_ThirdHyID = --.......
FROM [EnterPrisePositions]
WHERE ID = @PositionRows AND DATEDIFF(hour, ReleaseDate, GETDATE()) <= 24 * @ReleaseDateRange IF @@ROWCOUNT <> 0
BEGIN
SET @Salary_Score = NULL
SET @JobLocation_Score = NULL
SET @Education_Score = NULL
SET @Profession_Score = NULL
SET @Industry_Score = NULL
SET @JobNature_Score = NULL
SET @JobType_Score = NULL
SET @Computer_Score = NULL
SET @Language_Score = NULL
SET @CurrentValue = 0
SET @WeightSummary = 0
SET @ParaValue = NULL
SET @CurrentScore = NULL --计算薪资 SELECT @ParaValue = [Value] FROM [Parameters] WHERE [Type] = 'Salary' AND [Set1] = @Pos_Salary AND [Set2] = @Salary
IF @ParaValue IS NOT NULL
BEGIN
SET @Salary_Score = ABS(10 - @ParaValue)*@Salary_Weight
END IF @ProvinceId <> ''
BEGIN
IF (LEFT(@Pos_ProvinceId,4) = LEFT(@ProvinceId, 4) OR (
LEFT(@Pos_ProvinceId,2) = LEFT(@ProvinceId, 2) AND LEFT(@Pos_ProvinceId,2) IN ('','','','')))
AND @JobLocation_Same_City IS NOT NULL
BEGIN
SET @JobLocation_Score = ABS(10 - @JobLocation_Same_City)*@JobLocation_Weight
END
ELSE IF LEFT(@Pos_ProvinceId,2) = LEFT(@ProvinceId, 2) AND @JobLocation_Same_Province IS NOT NULL
BEGIN
SET @JobLocation_Score = ABS(10 - @JobLocation_Same_Province)*@JobLocation_Weight
END
ELSE IF @JobLocation_Not_Same IS NOT NULL
BEGIN
SET @JobLocation_Score = ABS(10 - @JobLocation_Not_Same)*@JobLocation_Weight
END
END --计算学历
IF @Education NOT IN ('', '不限')
BEGIN
IF @Pos_Education = @Education
BEGIN
SET @Education_Score = ABS(10 - @Education_Same) * @Education_Weight
END
ELSE
BEGIN
SET @Education_Score = ABS(10 - @Education_Not_Same)*@Education_Weight
END
END --计算专业
IF @Zydm <> ''
BEGIN
DELETE FROM @PositionSpecialty
INSERT INTO @PositionSpecialty (ID, Specialty)
SELECT ID, Specialty FROM [GetPositionSpecialty](@Pos_SpecialtyIds)
SELECT @Specialty_Rows = Count(*) FROM @PositionSpecialty
WHILE @Specialty_Rows <> 0
BEGIN
SELECT @Pos_SpecialtyId = Specialty FROM @PositionSpecialty WHERE ID = @Specialty_Rows
IF @Pos_SpecialtyId = '' OR LEFT(@Pos_SpecialtyId,2) = LEFT(@Zydm,2)
BEGIN
SET @Profession_Score = ABS(10 - @Profession_Match) * @Profession_Weight
END
ELSE
BEGIN
IF @Profession_Score <> 0 OR @Profession_Score IS NULL
SET @Profession_Score = ABS(10 - @Profession_Not_Match) * @Profession_Weight
END
SET @Specialty_Rows = @Specialty_Rows - 1
END END --其他略 --计算匹配度,加到匹配度临时表
IF @Salary_Score IS NOT NULL
BEGIN
SET @CurrentValue = @CurrentValue + @Salary_Score
SET @WeightSummary = @WeightSummary + @Salary_Weight
END
IF @JobLocation_Score IS NOT NULL
BEGIN
SET @CurrentValue = @CurrentValue + @JobLocation_Score
SET @WeightSummary = @WeightSummary + @JobLocation_Weight
END
IF @Education_Score IS NOT NULL
BEGIN
SET @CurrentValue = @CurrentValue + @Education_Score
SET @WeightSummary = @WeightSummary + @Education_Weight
END IF @WeightSummary != 0
BEGIN
SET @CurrentScore = @CurrentValue / @WeightSummary
SET @ParaValue = NULL
SELECT @ParaValue = [Score] FROM @PositionScore WHERE ID = @PositionRows
BEGIN
IF @ParaValue IS NULL
BEGIN
INSERT INTO @PositionScore(ID, Score, Salary_Score, Province_Score, Education_Score , Profession_Score , Industry_Score , JobNature_Score , JobType_Score , Computer_Score , Language_Score)
VALUES (@PositionRows, @CurrentScore, @Salary_Score, @JobLocation_Score,@Education_Score , @Profession_Score , @Industry_Score , @JobNature_Score , @JobType_Score , @Computer_Score , @Language_Score)
END
ELSE
BEGIN
IF @CurrentScore < @ParaValue
BEGIN
UPDATE @PositionScore SET Score = @CurrentScore WHERE ID = @PositionRows
END
END
END
END
END
SET @PositionRows = @PositionRows - 1
END --SELECT @ProvinceId, @Zydm, @JobNature, @JobTypeID, @SubJobTypeID,@Salary, @Computer, @Language , @Education, @HyID, @SubHyID, @ThirdHyID
SET @TableRows = @TableRows - 1
END
END
ELSE --学生没有建立职位搜索器
BEGIN
SELECT @Top1_Industry = [Value] FROM [Parameters] WITH (SNAPSHOT) WHERE [Type] = 'Industry' AND [Set1] = 'Top1'
SELECT @Top2_Industry = [Value] FROM [Parameters] WITH (SNAPSHOT) WHERE [Type] = 'Industry' AND [Set1] = 'Top2'
--.......
SELECT @Enterprise_Weight = [Value] FROM [Parameters] WITH (SNAPSHOT) WHERE [Type] = 'Weight' AND [Set1] = 'HasNoSearch' AND [Set2] = 'Enterprise' SELECT @Education = (CASE Xldm WHEN '' THEN '研究生' WHEN '' THEN '研究生' WHEN '' THEN '本科' WHEN '' THEN '高职' ELSE '' END), @Zydm = Gzydm
FROM .......[dbo].[StudentBasic] WITH (SNAPSHOT)
WHERE Xsxh = @Xsxh WHILE @PositionRows != 0
BEGIN --基本处理方法同前,也是一个个项目 SELECT @Pos_SubHyID = [SubHyID], @Pos_Education = [Education], @Pos_SpecialtyIds = [SpecialtyIds], @Pos_Zzjgdm = [OrgCode]
FROM [EnterPrisePositions] WITH(SNAPSHOT)
WHERE ID = @PositionRows
AND DATEDIFF(hour, ReleaseDate, GETDATE()) <= 24 * @ReleaseDateRange IF @@ROWCOUNT <> 0
BEGIN
SET @Education_Score = NULL
SET @Profession_Score = NULL
SET @Industry_Score = NULL
SET @Enterprise_Score = NULL --.......... IF @WeightSummary != 0
BEGIN
SET @CurrentScore = @CurrentValue / @WeightSummary
SET @ParaValue = NULL
SELECT @ParaValue = [Score] FROM @PositionScore WHERE ID = @PositionRows
BEGIN
IF @ParaValue IS NULL
BEGIN
INSERT INTO @PositionScore(ID, Score, Education_Score , Profession_Score , Industry_Score , Enterprise_Score)
VALUES (@PositionRows, @CurrentScore, @Education_Score , @Profession_Score , @Industry_Score , @Enterprise_Score)
END
ELSE
BEGIN
IF @CurrentScore < @ParaValue
BEGIN
UPDATE @PositionScore SET Score = @CurrentScore WHERE ID = @PositionRows
END
END
END
END
END SET @PositionRows = @PositionRows - 1
END --SELECT @Education, @Zydm END --根据各种输入参数输出
IF @ResultType = 1
SELECT s.*, p.*
FROM [EnterPrisePositions] P WITH (SNAPSHOT) LEFT OUTER JOIN @PositionScore S ON p.ID = s.ID
WHERE s.Score IS NOT NULL
ORDER BY Score
offset @PageSize*@StartPage rows fetch next @PageSize rows only --Sql2012的新的分页特性,效率很高
ELSE IF @ResultType = 2
BEGIN
if exists(select * from sysobjects where id=object_id('UserPositionResult'))
DROP TABLE UserPositionResult CREATE TABLE UserPositionResult
(
[ID] [int] IDENTITY(1,1) NOT NULL Primary Key NONCLUSTERED HASH WITH (BUCKET_COUNT = 4096),
[Score] FLOAT,
[PosiID] [uniqueidentifier] NOT NULL,
[PosiName] [nvarchar](40) NULL,
[EntName] [nvarchar](80) NOT NULL,
--...... WHERE s.Score IS NOT NULL
ORDER BY Score
END
ELSE
SELECT s.Score, p.PosiID, p.PosiName, p.EntName, p.EntUserID, p.JobNature, p.Number, p.Salary, p.Education, p.Specialty
FROM [EnterPrisePositions] P WITH (SNAPSHOT) LEFT OUTER JOIN @PositionScore S ON p.ID = s.ID
WHERE s.Score IS NOT NULL
ORDER BY Score
offset @PageSize*@StartPage rows fetch next @PageSize rows only --输出总记录数
SELECT Count(*) AS [TotalCount] FROM [EnterPrisePositions] P WITH (SNAPSHOT) LEFT OUTER JOIN @PositionScore S ON p.ID = s.ID
WHERE s.Score IS NOT NULL
END
GO
Sqlserver内存表的表现还是令人满意的,所有的职位匹配都计算一遍1秒还不到。想想哈希桶的威力是比较大,以前是B树索引,现在直接把时间复杂度近似降低到了O(1),对再大的数据量也是如此(需要设置合适的哈希桶数值)。
计算时的截图:
加大数据样本,到了上万条,也是1秒钟搞定。
心得:
- 对于不需要持久化的数据库,Sqlserver的内存表是最佳选择,建表的时候使用DURABILITY = SCHEMA_ONLY选项,不写日志,读写速度扛扛的;
- 长的存储过程,千万不要使用游标,性能极其低下,而且或产生一大堆的锁,影响其他进程。(改用While循环);
- 对内存表读取使用SnapShot隔离级别;对普通读取,实时数据准确度要求不高的情况下(比如数据分析)使用nolock隔离级别。
前台页面显示样式如下:
我们还做了邮件推送,定期给学生推送职位。
希望这篇文章起到抛砖引玉的作用,能够听取大家的建议。
基于海明距离的加权平均值人职匹配模型(Sqlserver2014/16内存表实现)的更多相关文章
- SpringBoot入门教程(五)Java基于MySQL实现附近的人
“附近的人”这个功能估计都不陌生,与之类似的功能最开始是在各大地图应用上接触过,比如搜附近的电影院,附近的超市等等.然而真正让附近的人火遍大江南北的应该是微信"附近的人"这个功能, ...
- 【佛山市选2013】JZOJ2020年8月7日提高组T3 海明距离
[佛山市选2013]JZOJ2020年8月7日提高组T3 海明距离 题目 描述 对于二进制串a,b,他们之间的海明距离是指两个串异或之后串中1的个数.异或的规则为: 0 XOR 0 = 0 1 XOR ...
- 海量数据相似度计算之simhash和海明距离
通过 采集系统 我们采集了大量文本数据,但是文本中有很多重复数据影响我们对于结果的分析.分析前我们需要对这些数据去除重复,如何选择和设计文本的去重算法?常见的有余弦夹角算法.欧式距离.Jaccard相 ...
- 64. 海明距离(Hamming Distance)
[本文链接] http://www.cnblogs.com/hellogiser/p/hamming-distance.html [介绍] 在信息领域,两个长度相等的字符串的海明距离是在相同位置上不同 ...
- 在线白板,基于socket.io的多人在线协作工具
首发:个人博客,更新&纠错&回复 是昨天这篇博文留的尾巴,socket.io库的使用练习,成品地址在这里. 代码已经上传到github,传送门.可以开俩浏览器看效果. 现实意义是俩人在 ...
- Matlab计算两集合间的海明距离
一.问题描述 B1[1 2 3 4 5 6 7 8 9] B2[12 13 14 21 31 41 51 1 1 81 1 1] 两个十进制矩阵,行数不一样,分别是n1和n2,列数必须一致,为nwo ...
- 使用simhash以及海明距离判断内容相似程度
算法简介 SimHash也即相似hash,是一类特殊的信息指纹,常用来比较文章的相似度,与传统hash相比,传统hash只负责将原始内容尽量随机的映射为一个特征值,并保证相同的内容一定具有相同的特征值 ...
- python海明距离 - 5IVI4I_I_60Y的日志 - 网易博客
python海明距离 - 5IVI4I_I_60Y的日志 - 网易博客 python海明距离 2009-10-01 09:50:41| 分类: Python | 标签: |举报 |字号大中小 ...
- 基于PaddlePaddle的语义匹配模型DAM,让聊天机器人实现完美回复 |
来源商业新知网,原标题:让聊天机器人完美回复 | 基于PaddlePaddle的语义匹配模型DAM 语义匹配 语义匹配是NLP的一项重要应用.无论是问答系统.对话系统还是智能客服,都可以认为是问题和回 ...
随机推荐
- C#对图片的操作
1.根据图片路径返回字节 public static byte[] getImageByte(string imagePath) { FileStream files = new FileStream ...
- 移动端lCalendar纯原生js日期时间选择器
网上找过很多的移动端基于zepto或jquery的日期选择器,在实际产品中也用过一两种,觉得都不太尽如人意,后来果断选择了H5自己的日期input表单,觉得还可以,至少不用引用第三方插件了,性能也不错 ...
- *HDU3486 RMQ+二分
Interviewe Time Limit: 2000/1000 MS (Java/Others) Memory Limit: 65536/32768 K (Java/Others)Total ...
- *HDU1285 拓扑排序
确定比赛名次 Time Limit: 2000/1000 MS (Java/Others) Memory Limit: 65536/32768 K (Java/Others)Total Subm ...
- PHP中获取星期的几种方法
PHP中获取星期的几种方法 PHP星期几获取代码: 1 date(l); 2 //data就可以获取英文的星期比如Sunday 3 date(w); 4 //这个可以获取数字星期比如123,注意0 ...
- Vim,极简使用教程,让你瞬间脱离键鼠切换的痛苦
注:看大家对Vim仇恨极大,其实它只是一种文本操作方式,可以减少键鼠的切换,从而让编辑文本的操作更迅捷.并不等同于IDE,在我看来,它们是两个是包含关系,IDE可以有Vim编辑模式.Vim或许可以通过 ...
- mvc配合jquery.validate验证失效,情况之一
用viewbage绑定input空间的value值,通过submit提交.validate验证失效. 1.应该是mvc的渲染顺序导致js验证失败. 解决方案:改用mvc自带的@html辅助方法,生成文 ...
- SqlServer删除表中重复记录
重复记录:有两个意义上的重复记录 一是完全重复的记录,也即所有字段均重复的记录: 二是部分关键字段重复的记录,比如Name字段重复,而其他字段不一定重复或都重复可以忽略. 1.对于第一种重复,比较容易 ...
- 解决警告“ld: warning: directory not found for option
因为已经把文件编译到项目中,删除的话会出现找不到文件或文件夹的警告. 1选择工程, 编译的 (targets) 2选择 Build Settings 菜单 3查找 Library Search Pat ...
- WPF打包32位和64位程序 运行在ghost WIN7上问题
WIN10,VS2015,编译平台"anycpu".WIN7系统为ghost版 1. 在.NET4.5下编译,程序打包以后,在WIN7上运行界面启动有3.4秒的延迟:将.NET版本 ...